The API

The Short Story

The short story is that PointsStalker can now update the Athletes and Points database on its own. PointsStalker checks for new databases at startup and will download the updated data in the background. The data associated with Groups will be kept, so updates should be unnoticeable aside from new data being presented when viewing an athlete. 

The Long Story 

Building the API server was a longer path of discovery than I expected. This was my first time building any kind of web service and I learned a lot. 

The API Basics

I decided to build the API on Sinatra. I liked Sinatra because it felt straight forward, all the routes are defined in one file, but still offers a lot of flexibility because it is built on top of Rack. Everything in one file; sounds simple right? Well, it is, but considering this was my first project in Ruby I had some catching up to do.

I felt it went well though, I did a few Ruby, Sinatra and Ruby on Rails tutorials and I jumped in. The basics came pretty easily, but, as with everything, the devil is in the details (gem file conflicts, configuring middleware, deploying to Heroku, etc.).

How the API Works  

Since it's my first time writing both iOS apps and server APIs I wanted to keep each component as basic and modular as possible. In particular, I figured I could always add functionality or complexity in the future, but I wanted to avoid getting stuck trying to do too much (or as my Dad used to say, 'come in hot and run out of talent'). So, the API is designed to deliver a JSON hash of the most recent database information the API is aware of for that version of the iOS app. Such data includes the list ID number, the date the list is valid, the date the list is valid to and, most importantly, a download link for this most recent database.

Upon receiving the latest database information for the API the iOS app will compare the JSON information to the information currently held on the phone and if the database held by the API is more recent then the iOS app will download the new SQLite database to replace the out of date information. 

Connecting to the API

When I started working on PointsStalker I had plans for how to approach every component of the app — including backup plans incase I got in over my head — every component except for one, networking. I knew almost nothing about networking from a programming perspective, yet it was critical to PointsStalker's overall functionality. So, in the process of reading the thousands of StackOverflow posts and listening to hundreds of developer podcasts while developing PointsStalker I always kept networking in the back of my mind. And at some point along the way I heard Marco Arment talking about using AFNetworking, so I kept it in mind for later. 

As it turns out, I probably shouldn't have been so concerned about networking. The internet is part of the foundation of nearly every single app created today, thus the tools must to be well developed and easily understood. It was incredibly simple to set up AFNetworking, make my HTTP GET requests and parse the JSON responses. It was even easier to download a link to a file in my S3 bucket, but my bubble had to burst at some point.

Configuring Background Fetch and Background Downloads

I fell back to earth while trying to configure the iOS background fetch API. The fetch API wasn't an issue, just figure out how to use the completion handler and implement application:performFetchWithCompletionHandler: in the app delegate. The problem became handling background downloads.

Again, the iOS API for handling background NSURLSessions was easy enough to understand. My difficulties stemmed from how to configure AFNetworking to use a NSURLSessionConfiguration for background downloading (which seemed more difficult than I felt it would to be). Eventually I got it figured out, but either I missed something or AFNetworking doesn't have a convenient way to configure download request for background downloading using the standard download methods. If you're in search of a solution check out this StackOverflow answer. 

The Result

With the API server up and running and the app successfully functioning with background API fetching and background downloads I feel good. There are still some bugs I need to try to iron out, but I'd feel pretty good about shipping what I have today. 

Next

In Progress: 

• UI Tweaks

Bugs Identified:

• DOB 1 day off for some people
• Database status string doesn't update
• Points graph legend gets compressed when using 'cursor'

Feature Updates

Feature Additions

1. User Created Groups / Lists 

I'm not sure what I want to call it yet, but user curated groups (lists?) currently function as follows. When you are in an athlete's detail view and tap the '+' button at the top right a group selection view will be displayed where you can see all your current groups, and if the athlete you were viewing is in one of those groups a check mark will be next to that group. To add the athlete to a new group or 

2. AutoLayout and iPhone 6 and 6 Plus Support

These aren't major additions for the app because most of my views are table views and text, so they scale with screen dimensions very easily, but I did completely redo the AthleteDetailView with AutoLayout to more easily support larger screen sizes and hopefully make it easier to support landscape orientation in the future (no promises though). 

Bug Fixes

1. Core Data - Read-Only SQLite

I don't fully understand all of the details, but from what I've gathered copying a pre-generated SQLite file from the app bundle to the user documents sandbox doesn't easily handle the multiple files generated while using Write Ahead Logging (WAL), and as a consequence Core Data doesn't like to open SQLite files that were generated with WAL as Read-Only. The most direct solution was to disable the WAL features for BOTH the importer and the app by including the following options string in the NSPersistentStore initialization. 

NSDictionary *options = @{NSSQLitePragmasOption:@{@"journal_mode":@"DELETE"}};

2. Search Crashes

There was a bug that caused the app to crash when you searched>View an athlete>return to the search>started to delete the search string one character at a time. The bug was caused 

Features In Progress

• Server-side Database Loading

Identified Bugs

• User curated groups don't persist after app restarts
• Lines through search results when changing search criteria after viewing an athlete from previous search criteria if the search results didn't fill the display. 

Core Data Import Performance

When creating the PointsStalker data importer I started by following a few different importing tutorials and eventually I got it all figured out. As the datasets I was importing grew from my test batch sizes grew from a few thousand to 18,000+ I realized I was having some scaling problems. The rough numbers said that the import speed was decreasing by a factor of 40x to 60x, so the complete imports were taking from 8 to 10 minutes for 18,000 athletes and their points histories. 

After a few months of tolerating the problem I decided to dig into the issue. Since the initial speed was reasonably good I ruled out the import logic as a major problem, but since the performance degraded over time I thought the problem might have been related to how data was being loaded into memory. In response I did some unscientific parametric testing of the data buffer size and the core data save interval, but the various values had nearly no effect on the overall processing time, so I started to look into other potential problems. 

The next point of interest was an NSMutableDictionary that is used to look up the managed object ID for athletes as new points list data is imported. I thought lookups might be taking a disproportionate amount of time because there are so many athletes to search though, but the points importing was in fact faster than athlete importing despite the points import logic being more complicated.

My gut feeling about NSMutableDictionary lookups was confirmed by an Objc.io article that indicated dictionary lookups should usually be O(1) (constant time). The article also mentioned hashing and creating dictionaries with pre-defined capacity, but concluded that +dictionaryWithCapacity was ineffective. This got me thinking about the performance of adding objects to the NSMutableDictionary and with a little searching I turned up a discussion on Cocoabuilder that explains that as the pre-determined dictionary size is filled to 67% capacity the dictionary is re-hashed and can impact performance. To avoid this the discussion suggests creating the NSMutableDictionary with the underlying CFDictionary class to create a pre-sized dictionary using:

NSMutableDictionary *dict = (NSMutableDictionary*)CFDictionaryCreateMutable(NULL, 1024, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks);

This change alone cut the import time to 1.5 minutes. To me, that's a huge decrease! Before the CFDictionary change I was wary of making changes that would required regenerating the database, but having MUCH shorter imports has removed that concern. 

(Maybe now I should go back and check out those data buffers and save interval parameters again now that I've eliminated a major limiting factor)